Three-dimensional computer animated actors may be useful in a variety of applications. For example, they may be employed for entertainment or educational purposes in film and television, where advanced techniques in 3-D modeling, kinematics and rendering allow the creation of realistic-looking action without the constraints of filming real actors. Computer animated actors may also play an integral role in 3-D video games and virtual reality, where they may help to achieve the goal of synthesizing realistic, interactive 3-D worlds.
Computer-animated actors may also be useful as a communication tool. In computer user interfaces, communication between the computer and the user may be carried out mainly through text. Instead, the computer may use an animated actor to communicate with the user. This may be accomplished by generating voice from the text using text-to-speech synthesis, while synchronizing the movements of the actor's face with the synthetic voice, by matching the pose of the face to the current sound. Substituting text with a humanoid talking actor may give the user a more personal, entertaining and engaging experience, and may reduce user fatigue caused by reading. Such a text-driven animated actor may be added to any application that relies on text, including web-based applications.
An important aspect of animating humans by computer is capturing the subtle and complex structure and movement of the human face. The face is commonly the focus of viewers' attention, especially during close-ups and when actors are speaking, and people are innately sensitive to even very small changes in expression. Therefore, accurately modeling and animating the human face may be viewed as a critical objective within the broader field of 3-D human animation.
Techniques for 3-D computer facial modeling and animation are reviewed in F. I. Parke and K. Waters, Computer Facial Animation, A. K. Peters, Wellesley, Mass., 1996, and in J. Noh, “A survey of facial modeling and animation techniques,” University of Southern California Technical Report 99-705, 1998. A 3-D model of a face may be developed using a variety of surface representations, such as, for example, polygonal or parametric surfaces. A polygonal surface is composed of a set of polygonal facets, such as triangles, joined at the edges. Parametric surfaces are composed from bivariate spline functions, also known as spline “patches.”
Realistic 3-D models of faces may be acquired readily from live subjects through various shape measurement techniques involving the use of active sensing, which casts special illumination onto an object in order to measure it. (For details on shape measurement by active sensing, see Y. F. Wang and J. K. Aggarwal “An overview of geometric modeling using active sensing”, in IEEE Control Systems Magazine, vol. 8, no. 3, pp. 5-13, 1988.) A variety of commercial shape capture systems using active sensing may be available, such as the 3030RGB/PS laser scanner of Cyberware Inc., Monterey, Calif.; the ShapeSnatcher light system of Eyetronics Inc., Belgium; or the 3DFlash! light system of 3DMetrics, Inc., Petaluma, Calif.
While accurate static models of faces may be readily and automatically acquired, animating the models realistically may be less straightforward. The task may involve determining appropriate deformations of the model. To limit the problem, a small set of reusable deformation procedures may be designed, which may be handled conveniently by a human animator or by an external program to generate deformations. An appropriate set of deformation procedures may simulate natural muscle movements of the human face. These muscle-like deformation procedures may be used in combination to simulate complex activities such as speech and emotional expression. The task of generating realistic facial animation thus may reduce to the task of designing a set of realistic muscle-like deformation procedures.
Procedures for muscle-like deformation of 3-D facial models may be classified into the following types: force propagation, displacement propagation, free-form deformation and direct surface displacement.
In a force propagation scheme, a facial model may include a representation of facial anatomy including elements corresponding to skin, muscle and bone. For the skin representation, multiple layers of skin tissue may be represented by an elastic spring lattice or a finite element model. For muscles, each muscle fiber may be represented as a vector between a skin node and an immobile bone attachment. Contraction of the muscle fiber results in pulling the skin attachment in the direction of the bone attachment. The force applied to one skin node is then propagated across the face through the skin tissue model.
This approach to facial deformation may require a great deal of data to reconstruct the complex underlying anatomy of the face and its physical properties, which may vary across features of the face. This may make such models painstaking to design. Furthermore, to compute the propagation of muscle contraction forces throughout the model may be computationally expensive.
To generate muscle-like deformation with less in-depth modeling and lighter computation loads, surface deformations may be computed more directly, without attempting to reconstruct the complex underlying anatomy and physical processes that lead to the deformations. Examples of these more result-oriented deformation control schemes may include the displacement propagation, free-form deformation and direct surface displacement methods.
A displacement propagation approach represents skin as an infinitesimally thin surface, with muscle fibers represented by vectors beneath the skin surface. Each vector has one moveable endpoint and one fixed endpoint. To simulate muscle contraction, the moveable endpoint of the vector moves in the direction of the fixed endpoint. As the moveable endpoint is displaced toward the fixed endpoint, control points on the skin surface within a zone of influence of the muscle vector are also displaced in the direction of the fixed endpoint. The magnitude of the displacement of each control point in the zone of influence may be a function of its angular distance from the muscle vector and its nearness to the immobile endpoint. The magnitude of displacement may also be affected by a skin elasticity factor.
In free-form deformation, a surface is deformed by manipulating an invisible, flexible bounding box in which the surface is embedded. As the bounding box is deformed by manipulating its control points, the embedded surface deforms accordingly. Free-form deformation may be used to simulate muscle-like actions by displacing control points of a bounding box along particular trajectories.
In both of the two preceding techniques—displacement propagation and free-form deformation—a facial model involves a simple surface controlled by the displacement of secondary structures, whether muscle vectors or a bounding box. On the other hand, in the direct surface displacement method, the displacement of the surface is described directly, not as a function of the displacement of some other structure. The displacement of a group of control points in the surface may be described by a parametric equation, for example.
While these three method—displacement propagation, free-form deformation and direct surface displacement—all may involve less complex models and less intensive computation than the force propagation method, such deformation schemes may nevertheless require significant painstaking effort to design. In each case, it may be necessary to specify the various data and functions that will engender the desired surface deformations. For example, in the displacement propagation approach, one may be required to specify the placement and zone of influence of each muscle vector, and possibly an elasticity parameter over the skin surface; in free-form deformation, the bounding box may need to be specified as well as the displacement trajectories of its control points; and for direct surface displacement, one may be required to design the equations to govern the trajectories of the control points. These data may need to be supplied by artistic interpretation of facial deformations. Furthermore, the greater the desired realism and accuracy of the deformations, the more labor and skill may be required to specify the deformation procedures.